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ABSTRACT 


This  report  assesses  the  documentation  of  the  Department  of  Energy's  Midterm 
Oil  and  Gas  Supply  Model.  The  objective  here  is  not  merely  to  assess  docu- 
mentation, but  also  to  present  a method  for  documentation  assessment.  This 
investigation  has  resulted  in  guidelines  which  can  be  used  both  to  assist  pro- 
ject sponsors  in  determining  their  documentation  needs,  and  as  a standard 
against  which  to  compare  existing  model  reports.  The  documentation  guidelines 
presented  here  amplify  but  do  not  alter  substantively  the  DOE  "Interim  Model 
Guidelines"  of  December  1978  and  are  organized  according  to  levels  associated 
with  the  information  needs  of  various  phases  of  model  operation.  The  documen- 
tation of  the  Midterm  Oil  and  Gas  Supply  Model  is  discussed  in  light  of  these 
guidelines.  The  guidelines  are  recommended  for  incorporation  into  DOE  model 
development  projects. 
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1.  INTRODUCTION 


This  first  interim  report  from  the  National  Bureau  of  Standards  (MBS)  project 
for  "Energy  Model  Validation  Procedure  Development"  documents  our  performance 
of  Task  2 of  the  project  work  statement  (see  Appendix  A),  which  requires  the 
assessment  of  the  documentation  of  the  Midterm  Oil  and  Gas  Supply  Model.  This 
report  records  that  assessment  and  presents  a method  for  the  assessment  of 
model  documentation  as  an  independent  step  in  model  assessment.  Thus,  this 
report  is  an  initial  response  to  the  generic  concerns  listed  in  our  work 
statement;  viz.,  "to  develop  and  apply  standards  and  procedures  for  the  vali- 
dation of  analysis  systems  utilized  by  the  Energy  Information  Agency  (EIA)  of 
the  DOE." 

We  approached  both  our  task  2 and  the  generic  methodology-development  task 
from  the  viewpoint  of  model  assessors.  We  sought  answers  to  the  following 
basic  questions  which  arise  in  the  assessment  of  a model. 

1)  What  was  the  model  supposed  to  be?  (Documentation  accompanying  the 
model  is  the  only  proper  source  of  such  information. ) 

2)  What  did  the  model  turn  out  to  be?  (Computer  code  is  a necessary  but 
not  sufficient  source  for  this  information.) 

3)  Is  the  resulting  form  consistent  with  the  intent? 

4)  What  are  the  alternatives  to  this  model? 

5)  What  are  the  relative  strengths  and  weaknesses  of  the  model  and  its 
alternatives? 

^Sponsored  by  the  Department  of  Energy,  Office  of  Analysis  Oversight  and  Ac- 
cess, Interagency  Agreement  No.  EA  77-A-O 1-6610 
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Task  2 is  concerned  with  establishing  a sufficient  understanding  of  the  code 
and  the  conceptual  model  for  us  to  answer  questions  1,  2,  and  3 above. 
Questions  4 and  5 are  clearly  important  but  outside  the  central  scope  of  task 
2.  They  will  be  addressed  in  later  reports. 

Initially,  our  plan  was  to  examine  the  documentation  of  the  conceptual  model 
and  its  implementing  computer  programs,  so  as  to  understand  the  model  and  its 
outputs,  and  to  set  up  and  execute  the  computer  programs  with  several  sce- 
narios. Unfortunately,  the  set  of  documents  transmitted  to  us  by  DOE  (see 
Appendix  B)  did  not  contain  sufficient  information  to  accomplish  this  plan. 

We  concluded  that  it  would  not  be  possible  to  complete  the  assessment  project 
without  additional  documentation.  Therefore,  to  make  it  possible  to  achieve 
the  research  objectives  of  the  project,  we  expanded  our  source  materials  by 
identifying  and  acquiring  several  ancillary  documents;  Appendix  C contains  a 
list  of  the  documents  we  obtained. 

Because  this  material  was  not  organized  in  any  concise  or  cohesive  manner,  its 
assimilation  became  a stumbling  block.  When  augmented  by  information  obtained 
from  consultations  with  the  model  developers  and  DOE  staff,  the  material  pro- 
vided us  with  sufficient  understanding  of  the  conceptual  model  and  its  real- 
ization to  allow  the  assessment  project  to  proceed.  We  judged  that  by  itself 
the  expanded  list  of  documentation  would  have  imposed  inordinate  effort  and 


time  on  the  task  of  model  assessment. 
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Our  documentation  assessment  efforts  were  aided  greatly  by  previously  suggest- 
ed guidelines  for  model  and  computer  program  documentation  [1,2,5].  Thus,  our 
initial  documentation  assessment  approach  was  simply  to  examine  the  DOE  docu- 
ments and  review  them  in  light  of  these  earlier  documentation  guidelines.  We 
would  have  liked  to  have  performed  field-tests  to  determine  whether  documents 
that  conform  to  the  suggested  guidelines  were  sufficient  for  the  needs  of  mod- 
el users  and  of  model  assessors.  However,  the  information  in  the  DOE  docu- 
ments was  not  close  enough  to  the  information  requirements  of  the  suggested 
documents  for  this  to  be  done.  We  did  complete  a review  of  the  DOE  document- 
ation and  used  this  experience  to  modify  the  guidelines.  The  resultant  guide- 
lines have  not  been  tested  by  their  use  in  a model  development  project.  Even 
so,  we  do  feel  that  their  adoption  by  DOE  will  improve  the  value  of  document- 
ation and  the  utility  of  DOE  models. 

Before  proceeding  with  the  development  of  recommendations  concerning  specific 
documental  requirements,  we  wish  to  comment  on  the  importance  of  good  documen- 
tation and  on  the  practical  question  of  documentation  "workload."  It  is  gen- 
erally understood  that  the  preparation  of  proper  documentation  for  a large- 
scale  mathematical  model  is  at  best  difficult,  time  consuming,  and  expensive. 
Ironically,  production  of  explicit  descriptions  of  a model's  minutiae,  sup- 
porting analyses,  and  modes  of  operation  may  require  a more  comprehensive 
grasp  of  the  model's  principles  and  their  realization  than  does  the  model 
development  itself.  The  modeler  experiments  in  constructing  the  "final"  mod- 
el; the  documentor  does  not  have  the  luxury  of  experimentation.  Slight  mis- 
calculation in  the  construction  of  a model  might  escape  deleterious  conse- 
quences because  of  "compensating  errors"  or  its  submergence  below  the  "noise 
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level"  of  a process  or  data,  but  a flawed  description  of  a ” delivered"  model 
will  cause  serious  inconvenience  or  worse  for  users.  It  is  not  unusual  for 
this  to  be  so  -even  when  the  users  are  the  model  developers!  The  importance  of 
documentation  can  be  briefly  conveyed  in  terms  of  the  extreme  case  of  virtual- 
ly no  documentation.  Without  documentation,  a model  is  a black  box  to  which 
one  does  not  know  how  to  pose  questions  and  whose  responses  cannot  be  inter- 
preted fully.  Moreover,  if  documentation  of  the  computer  procedures  is  lack- 
ing, the  black  box  will  be  sealed  and  inert;  no  questions  can  be  asked  nor  any 
answered. 

The  sentiments  above  are  seldom  if  ever  disputed  in  principle,  but  models  con- 
tinue to  be  produced  wholesale,  accompanied  by  documentation  that  is  scanty, 
disorganized,  and  unclear  or  even  misleading. 

Why  is  this  so  in  spite  of  the  great  amount  of  conscientious  effort  in  model- 
ing projects?  Typically,  modeling  activity  takes  place  with  funding  restric- 
tions and  under  the  pressure  of  time  deadlines.  Project  sponsors  frequently 
demand  ad  hoc  exercise  of  models  still  under  formulation  for  "crash”  investig- 
ation of  special  questions,  or  they  prescribe  shifts  in  modeling  objectives  in 
midstream.  Modelers  complain  with  justification  that  they  do  not  have  time 
for  production  of  proper  documentation.  Moreover,  it  is  in  the  nature  of 
problems  which  require  complex  large-scale  models  for  attempts  at  solution, 
that  the  magnitude  of  associated  uncertainties  ensures  that  a "perfect”  or 
"finished"  model  can  never  be  achieved.  Given  this  unpleasant  circumstance, 
modelers  would  always  rather  expend  any  available  increment  of  effort  toward 
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refinement  of  the  model  than  for  documentation.  One  purpose  of  the  guidelines 
is  to  remind  modelers  that  there  is  a point  at  which  the  pursuit  of  the  will 
o’  the  wisp  of  perfection  must  be  temporarily  abandoned  in  favor  of  taking 
stock  and  preparing  signposts  to  allow  others  to  follow  the  trail. 

There  are  also  professional  considerations.  In  all  sciences  the  requirement 
to  elaborate  clearly  the  procedures  for  replication  of  experiments  or  analysis 
supporting  published  results  is  not  merely  good  practice.  It  is  a binding  ob- 
ligation imposed  by  the  standards  of  the  academic  community.  Moreover,  re- 
search models  are  customarily  developed  in  order  to  buttress  a particular 
statement  or  theory,  while  models  such  as  the  energy  models  considered  here 
will  in  general  be  used  many  times  by  their  developers  as  well  as  many  others. 
Thus,  models  produced  under  aegis  of  the  federal  government  are  first  of  all 
likely  to  be  subject  to  acceptance  criteria  analogous  to  those  for  various 
kinds  of  hardware  procurement,  i.e.  they  must  be  accompanied  by  information 
assuring  feasibility  of  use  and  maintenance.  If  models  are  intended  for  use 
as  decision  aids  in  support  of  non-trivial  policy  matters,  they  are  also  sub- 
ject to  outside  critical  scrutiny  by  "interested  parties,"  i.e.  any  groups  up- 
on whom  the  consequences  of  policy  decisions  may  have  real  effect.  In  fact, 
there  is  a statutory  requirement  for  public  accessibility  of  the  energy  models 
which  are  the  nominal  subject  of  this  report.  Although  accessibility  has  not 
yet  been  defined  beyond  quibble,  there  is  no  question  that  it  implies  open 
availability  of  sufficient  information  for  comprehension  of  the  model  and  its 


outputs 
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The  balance  of  this  report  contains  the  following  material:  Section  2 sunma 

r.zes  a general  methodology  for  documentation  assessment;  Section  3 presents 
an  evaluation  of  the  documentation  of  the  DOE’s  Midterm  Oil  and  Gas  Supply 
Model  using  this  methodology;  Section  4 presents  the  conclusions  drawn  from 
our  efforts  to  date;  Section  5 is  a bibliography.  The  appendices  contain,  r 
spectively,  the  work  statement  for  the  over-all  project,  the  memorandum  con- 
taining the  initial  list  of  documentation,  and  a list  of  the  documents  that 
were  evaluated. 
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2.  PHASE-STRUCTURED  DOCUMENTATION 

We  believe  that  the  currently  available  documentation  of  the  DOE  Midterm  Oil 
and  Gas  Supply  Model  does  not  contain  sufficient  information,  organized  in  a 
readily  accessible  format,  to  facilitate  model  assessment,  he  have  justified 
that  judgment  by  developing  and  applying  a formal  process  for  documentation 
assessment.  To  this  end  we  have  classified  model  operation  into  four  phases, 
each  identified  by  its  information  requirements  and  associated  documents. 

We  believe  that  the  documentation  required  to  use  a model  depends  on  the  im- 
portance and  complexity  of  problems  addressed  through  it.  For  example,  a 
simple  model  intended  to  mechanize  the  analysis  of  a noncontroversiai  system 
may  not  require  extensive  documentation.  In  fact,  in  many  cases,  the  full 
amount  of  necessary  information  can  exist  in  the  form  of  comment  cards  inter- 
spersed throughout  the  computer  code.  On  the  other  hand,  a large-scale  model 
used  for  analysis  supporting  national  policy  development,  and  expected  to  un- 
dergo extensive  critical  appraisal,  is  likely  to  require  a very  complete  set 
of  documentation.  Our  classification  scheme  is  a hierarchical  structure  in 
which  the  resulting  set  of  documents  for  each  phase  or  "level"  includes  all 
the  documentation  in  the  preceding  levels. 
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The  documentation  plan  presented  below  should  be  viewed  as  one  approach  to  or 
ganizing  an  acceptable  set  of  documentation.  Since  many  other  organizations 
[3,4]  are  possible,  one  need  not  evaluate  the  docunentation  of  a particular 
modeling  project  by  whether  or  not  each  of  the  designated  documents  exists, 
but  rather  by  whether  or  not  all  the  information  required  in  each  of  these  re 
ports  can  be  found  readily.* *  On  the  other  hand,  for  intercomparison  of  a col 
lection  of  models  the  benefits  of  a uniform  standard  for  the  organization  of 
model  documents  are  indisputable. 

2. 1 Phased  Documentation  Organization 


We  next  describe  the  four  phases  (or  levels)  of  documentation,  listing  the 
document  types  associated  with  each.  Detailed  descriptions  of  the  documents 
are  given  in  Section  2.2. 

2.1.1  Level  I:  Rote  Operation  of  the  Model 


Phase  one  is  concerned  with  requirements  for  rote  execution  of  computer  runs, 
i.e.  the  "ground  rules  for  setting  up  and  running  the  model"  on  a particular 
computer  and  for  verifying  the  correctness  of  the  execution.  The  document 
types  are: 


*Some  of  the  designated  documents  could  be  quite  brief,  particularly  if  they 
provide  references  to  open  literature  sources  of  greater  depth.  Moreover, 
those  reports  intended  to  be  tutorial  in  nature  could  be  incorporated  into 
their  technical  counterparts  as  introductions,  if  their  functions  are  clearly 
highlighted. 
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(1-1)  Operations  Manual, 

(1-2)  Data  Base  Description:  Physical/Logical  Characteristics,  and 

(1-3)  Software  Description:  User  Level. 

2.1.2  Level  II:  Model  Use 

Phase  two  provides  an  explanation  of  a given  set  of  scenarios  and  enables  con- 
struction of  new  scenarios  and  interpretation  of  this  output.  The  relevant 
documents  are  those  specified  in  I above  and: 

(II-l)  Mathematical  Description 

(II-2)  Data  Requirements  Report:  Sources,  Transformations,  and 

Justification,  and 
(II-3)  Process  Description. 

2.1.3  Level  III:  Model  Maintenance 

In  this  phase,  the  documentation  addresses  modification*  of  the  computer  code 
(and  perhaps,  therefore,  the  conceptual  model)  to  investigate  scenarios  which 
range  beyond  originally  conceived  limits  or  assumptions.  The  relevant  docu- 
ments are  those  specified  in  I and  II  above  and: 

(III-l)  Software  Description:  Programmer  Level,  and 

(III-2)  Maintenance  Log. 

*We  adopt  hereby  the  assumption  that  a "major"  change  in  a model's  structure 
ipso  facto  defines  a new  model,  which  would  require  new  documentation,  al- 
though this  could  be  produced  in  part  by  modification  of  original  documents. 


2.1.4  Level  IV:  Model  Assessment 


In  order  to  conduct  a third-party  assessment,  model  assessors  should  have  all 
model  documentation  available  to  them,  especially  during  the  documentation 
assessment  phase.  While  some  of  the  documents  listed  below  are  not  written 
specifically  with  assessors  in  mind  (e.g.  the  IV-2  is  aimed  at  the  policy 
maker),  they  are  invaluable  as  aids  in  understanding  how  a model  was,  and  is 
intended  to  be,  used.  However,  even  in  the  case  where  a sponsor  determines 
that  a model  is  not  to  be  assessed,  many  of  these  reports  should  still  be  pro- 
duced. In  this  case,  the  reports  would  be  assessed  in  terms  of  the  intended 
uses  outlined  in  sections  2.2.9  through  2.2.12.  The  criterion  for  acceptable 
documentation  is  modified  here  since  some  of  the  following  documents  may  not 
exist  either  because  the  work  they  describe  has  not  been  performed,  as  in  the 
case  of  (1)  and  (2)  below,  or  because  the  potential  benefits  of  the  report  to 
a particular  project  might  be  judged  insufficient  to  justify  their  creation, 
as  in  the  case  of  (3),  (4),  and  (5).  The  relevant  documents  are  those  speci- 
fied in  I,  II,  and  III  above  and: 

(IV-1)  Assessment  Report, 

(IV-2)  Model  Application  Report, 

(IV-3)  Model  Summary , 

(IV-4)  Historical  Record,  and 

(IV-5)  all  other  documents  written  about  the  model  and  not  specifically 


listed  above. 


-11- 


Note  that  the  hierarchy  of  levels  goes  from  I (lowest)  to  IV  (highest)  but 
that  the  order  of  listing  of  documents  within  a level  has  no  intended  signifi- 
cance. 

2. 2 Document-Type  Descriptions 


The  following  descriptions  of  the  documents  listed  above  have  evolved  from 
those  originally  presented  in  [1],  [2],  and  [5]  to  the  present  form  as  a re- 
sult of  the  partial  field-testing  performed  using  the  documentation  of  the 
Midterm  Oil  and  Gas  Supply  Model.  The  documents  are  described  here  in  the 
order  in  which  they  were  listed  above. 

2.2.1  Operations  Manual 

The  purpose  of  this  manual  is  to  provide  computer  operations  personnel  with  a 
description  of  the  software  and  of  the  operational  environments  so  that  the 
software  can  be  run.  The  operations  manual  should  include  an  overview  of  the 
software  organization,  the  program  and  file  inventory,  a list  of  the  kinds  of 
runs  possible,  a description  of  program  control  flow,  a list  of  the  run  stream 
control  statements,  estimated  run  time  and  turnaround  time  (at  a particular 
installation),  an  annotated  list  of  files  created  or  changed,  and  concise  in- 
formation on  size,  number,  and  type  of  output  reports.  For  more  information 
on  this  document,  see  [1]. 
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2.2.2  Data  Base  Description:  Physical/Logical  Characteristics 

In  [1]  this  document  was  called  Data  Base  Specification.  Its  purpose  is  to 
specify  the  identification,  logical  characteristics,  and  physical  character- 
istics of  the  data  base  for  the  model.  Highlights  of  the  Data  Base  Descrip- 
tion include  special  instructions  for  data  entry,  tape  and  file  labeling  con- 
ventions, support  software  description,  logical  characteristics  of  the  data 
(arrangement,  relationships),  and  physical  characteristics  (storage,  access). 
For  more  information,  see  [1]. 

if 

2.2.3  Software  Description:  User  Levelt 

The  purpose  of  this  manual  is  to  describe  the  functions  performed  by  the 
software  in  non-ADP  terminology,  so  that  the  user  organization  can  determine 
the  logistics  of  its  applicability  and  how  to  put  it  into  operation.  It 
should  serve  as  a reference  document  for  preparation  of  input  data  parameters 
and  for  interpretation  of  results.  For  more  on  this  report,  see  the  User’s 
Manual  in  [1]. 


-fr-  — — - - — — 

Titles  marked  with  a dagger  (t)  identify  documents  that  will  not  ordinarily 
require  revision  for  minor  modifications  of  a model. 


-13- 


2.2.4  Mathematical  Description 


This  report  describes  (a)  the  complete  details  of  the  mathematical/logical 
model,  including  assumptions  and  hypotheses,  (b)  the  rationale  for  its  form, 
including  some  discussion  of  alternatives,  and  (c)  restrictions  on  the  use  of 
the  model.  This  is  an  operational  document  maintained  throughout  the  model 
life  cycle.  Model  structures  are  frequently  modified  over  time  and  procedures 
for  updating  this  document  should  exist. 

2.2.5  Data  Requirements  Report:  Sources,  Transformations,  and  Justifications 

This  document  describes  the  detailed  data  needs  of  the  model  including  input 
variables  and  "hardwired"  parameters;  sources  for  all  data,  alternative  data 
sources,  if  any,  and  justifications  for  choice  of  data  sources;  processes  for 
obtaining  data  and  for  transforming  data  for  model  compatibility;  organiza- 
tional and  individual  responsibilities  for  obtaining,  updating,  and  processing 
data;  numerical  and  forecasting  techniques  to  be  used  for  estimation  of  input 
parameters;  data  consistency  checks;  and  acceptable  data  ranges.  This  is  an 
operational  document  to  be  maintained  throughout  the  lifespan  of  the  model. 

It  should  be  complete  in  that  it  should  give  references  and/or  justifications 
for  all  data;  but  it  can  reference  the  Mathematical  Description  for  more  de- 
tailed descriptions  of  the  data  uses. 

2.2.6  Process  Description! 

This  report  should  describe  the  background  of  the  problem  and  provide  at  least 
an  outline  description  of  the  underlying  physical,  economic,  technological, 
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and  behavioral  processes  to  be  modeled.  It  includes  a general  description  of 
the  problem  and  decision  environment,  provides  an  historical  perspective,  and 
describes  all  information  believed  significantly  relevant  to  the  decision- 
making process  regardless  of  whether  the  ultimate  model  includes  each  of  these 
topics.  The  reader  should  be  assumed  unfamiliar  with  the  topic  area  and, 
therefore,  the  report  should  provide  definitions  of  terminology  and  references 
to  other  source  documents  which  more  fully  expand  the  subject  areas  presented. 

2.2.7  Software  Description:  Programmer  Level 

The  purpose  of  this  report  is  to  provide  the  maintenance  programmer  with  the 
information  necessary  to  understand  the  programs,  their  operating  environment, 
and  their  maintenance  procedures.  In  particular,  if  the  model  is  to  be  used 
with  a computer  other  than  the  one  for  which  the  computer  programs  were  com- 
posed originally,  any  specific  hardware-related  restrictions  or  characteris- 
tics should  be  spelled  out.  For  more  on  this  document,  see  the  description  of 
the  Program  Maintenance  Manual  in  [2]. 

2.2.8  Maintenance  Log 

This  document  describes  the  process  of  maintaining  and  updating  the  model.  It 
is  a log  which  identifies  and  records  changes  made  to  the  model  and/or  its 
data,  and  from  which  can  be  extracted  an  "official”  audit  trail.  The  log 
should  contain  the  date  of  the  change,  the  persons  responsible  for  the  change, 
and  a brief  phrase  identifying  or  naming  the  change.  It  is  assumed  here  that 
a detailed  discussion  of  the  modifications,  including  the  reasons  for  and  the 
implications  of  these  changes,  will  be  included  in  the  documentation  of  either 
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mathematics  or  data.  When  this  detailed  description  is  available,  a reference 
to  it  should  be  appended  to  the  corresponding  log  entry.  This  log  should  be 
used  throughout  the  life  cycle  of  the  model,  beginning  in  the  formulative 
stages.  It  is  not  necessary  that  this  log  be  of  publishable  quality,  but  it 
is  important  that  it  be  kept. 

2.2.9  Assessment  Report 


This  report  includes  a description  of  any  model  assessment  plan  agreed  to  by 
the  user/sponsor  and  model  developer,  and  the  results  of  implementing  that 
plan.  It  should  include  all  tests  of  the  model  output  in  terms  of  comparisons 
to  historical  data,  acceptability  to  the  user  (on  the  basis  of  prior  experi- 
ence or  intuition),  statistical  measures,  and  comparative  results  of  altern- 
ative formulations.  The  developers  must  state  and  explain  deficiencies  and 
anomalies  of  the  model  output  as  well  as  agreements  with  expected  outcomes. 
This  report  should  at  least  delineate  the  problem  environment  in  which  the  mo- 
del is  known  to  produce  results  acceptable  to  the  user/sponsor,  and  those  en- 
vironments in  which  the  results  are  unacceptable.  The  plan  should  be  reexe- 
cuted whenever  the  model  is  changed,  and  this  report  should  be  updated  accord- 
ingly. 

2.2.10  Model  Application  Report 

This  report  describes  the  results  of  exercising  the  model  to  obtain  answers  to 
specific  questions  posed  by  decision-makers,  or  to  study  the  behavior  of  the 
problem  environment  as  it  is  represented  by  the  model.  It  is  directed  toward 
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the  executives  of  the  organization  who  will  use  the  interpreted  results  of  the 
model  to  make  decisions.  This  report  should  clearly  and  completely  describe 
the  scenario  being  modeled,  but  need  not  include  all  of  the  technical  details 
of  the  model,  the  consequences  of  these  assumptions,  and  the  capabilities  and 
limitations  of  the  model.  Input  parameters  and  assumptions  to  which  the  model 
results  are  particularly  sensitive  should  be  identified  explicitly.  To  the 
extent  possible,  this  document  should  quantify  changes  in  model  results  assoc- 
iated with  changes  in  these  sensitive  parameters  and  assumptions.  Much  of 
this  material  may  be  summarized  from  the  Mathematical  Description,  the  Data 
Requirements  Report,  and  the  Maintenance  Log.  The  Model  Application  Report 
should  be  of  publishable  quality  in  that  it  should  be  self-contained,  it 
should  define  terminology,  and  it  should  provide  references  to  supporting 
documents. 

2.2.11  Model  Summaryt 

This  report  is  a nontechnical  summary  of  the  basic  information  that  describes 
the  model.  Its  purpose  is  to  provide,  in  a concise  fashion,  a description  of 
the  model  to  a broad  audience  so  that  they  may  determine  if  it  is  of  interest 
to  them.  This  document  can  be  included  in  other  documents  or  may  be  distrib- 
uted separately. 

2.2.12  Historical  Record! 


This  report  describes  the  questions  or  problems  which  led  to  the  decision  that 
the  model  was  needed,  and  how  the  model  is  to  be  used  to  address  these  issues. 
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It  describes  the  procedures  used  to  determine  that  the  model  can  and  should  be 
developed,  including  a discussion  of  the  advantages  and  disadvantages  of  al- 
ternatives to  modeling,  and  at  least  summary  comparisons  to  alternative 
models*  The  document  should  describe  constraints  on  time,  funding,  personnel, 
and  computer  facilities,  that  could  significantly  affect  the  nature,  scope, 
and  approach  of  the  modeling  effort.  Major  participants  should  be  identified 
by  name,  technical  background,  professional  affiliation,  and  areas  of  respons- 
ibility. This  document  is  most  easily  prepared  if  it  is  written  in  the  earli- 
est stages  of  model  conception,  when  the  information  is  still  current  and  thus 
more  likely  to  be  complete.  It  can  be  constructed  from  memoranda,  meeting 
notes,  and  proposals,  and  need  not  be  a formal  publication.  It  should,  how- 
ever, be  complete  in  that  it  defines  terminology  and  references  any  sources  of 
information  pertinent  to  this  phase  of  model  development. 

2.2.13  Other  Documents 


It  is  a difficult  task  to  determine  which  kind  of  information  should  be  re- 
corded for  posterity  and  which  should  not.  The  levels-of-use  concept  intro- 
duced here  should  help  in  making  that  determination,  but  it  is  not  intended  to 
reduce  documentation  decisions  to  a trivial  exercise.  In  any  case  whichever 
documents  are  produced  should  be  made  available  to  the  assessment  team.  In 
addition  to  the  documentation  set  described  above,  there  are  other  documents 
which  could  be  useful,  and,  if  written,  should  be  given  to  assessors.  These 
reports,  with  which  we  do  not  associate  a high  priority,  might  include  soft- 
ware debugging  plans  and  results,  procedures  for  training  model  users,  and 
other  reports  identified  in  either  [1]  or  [2]  and  not  specifically  described 
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2. 3 Information  Organization  vs.  Laundry  Lists 

Neither  the  designation  of  phases  nor  the  specific  organization  of  information 
into  documents  should  be  regarded  as  attempts  to  establish  definitive  docu- 
mentation procedures;  they  were  selected  to  facilitate  exposition  of  inform- 
ation requirements  and  because  they  appeared  to  us  to  give  rise  to  a natural 
grouping  of  model  information  from  the  viewpoint  of  "decision-responsible”  mo- 
del clients.  Moreover,  the  list  of  documents  should  not  be  viewed  by  a model 
sponsor  as  guaranteeing  adequate  documentation  simply  by  requesting  production 
of  "one  of  each  of  those  reports."  Nor,  by  the  same  token,  can  the  developer 
feel  confident  of  providing  adequate  documentation  just  by  presenting  reports 
containing  more  boilerplate  than  substance,  which  are  nominally  appropriate  by 
virtue  of  bearing  the  listed  titles.  Instead,  model  sponsors  and  model  devel- 
opers should  be  concerned  with  developing  a documentation  plan  which  insures 
that  model  documentation  provides,  in  a well-organized  manner,  all  of  the  in- 
formation determined  to  be  appropriate  for  the  expected  level  of  use  of  a giv- 
en model.  These  two  constructs,  information  and  organization,  are  the  core  of 
proper  documentation,  and  are  of  equal  importance. 

By  information,  we  mean  the  document  content  which  is  elaborated  at  length 
throughout  this  report.  By  organization,  we  do  not  mean  merely  the  particular 
configuration  in  which  quantities  of  information  are  grouped  or  concatenated 
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(although  considerable  importance  should  be  attached  to  lucid  exposition  with- 
in any  given  document).  Indeed  we  classified  information  content  according  to 
a scheme  that  appeared  most  reasonable  to  us,  but  we  had  relatively  minor  res- 
ervations against  those  outlined  in  [3]  and  in  the  interim  guideline  [5].  The 
critically  important  aspect  of  organization  relates  to  the  ease  of  access  to 
information  by  a user.  This  means  that  (1)  the  materials  in  the  documents 
must  be  indexed  for  quick  reference  to  any  desired  item,  and  (2)  tables  of 
contents,  and  equivalently  section  headings,  should  be  carefully  worded  to 
provide  a usable  road  map  of  the  substance  of  the  documentation. 

We  believe  the  lists  of  information  requirements  presented  in  this  chapter  can 
be  used  as  a checklist  when  evaluating  model  documentation,  again  keeping  in 
mind  that  "proper  documentation  provides  specific  and  detailed  information 
that  is  organized  and  presented  in  a manner  that  will  satisfy  the  needs  of 
each  segment  of  a model’s  audience"  [2].  Furthermore,  we  feel  that  the  recom- 
mended documents  described  and  discussed  in  this  section  represent  a "com- 
plete" set  of  documentation  in  that  if  each  of  these  documents  is  produced  ac- 
cording to  accepted  professional  publication  standards,  they  will  provide  a 
prospective  user  institution  (with  a suitable  computer),  the  capability  for 
independent  analysis  and  experimentation  based  on  the  model.  We  will  thus 
have  moved  a long  way  toward  developing  useful  models  that  can  be  used. 
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3.  DOCUMENTATION  ASSESSMENT 

In  this  section,  we  evaluate  the  documentation  (see  Appendix  C)  of  the  Midterm 
Oil  and  Gas  Supply  Model  in  terms  of  the  guidelines  developed  in  Section  2. 
While  the  discussion  includes  the  results  of  comparing  the  information  pre- 
sented in  the  reports  in  Appendix  C to  that  of  the  ideals  defined  in  Section 
2,  it  should  not  be  viewed  as  a report  card,  because  the  state-of-the-art  of 
model  documentation  is  that  standards  are  still  under  development  and  defici- 
encies in  documentation  appear  to  be  universal.  Rather  it  provides  informa- 
tion we  feel  can  be  useful  in  planning  future  efforts.  In  fact,  since  DOE  is 
in  the  process  of  remedying  specific  documentation  deficiencies,  the  discus- 
sion below  might  be  used  to  help  evaluate  those  documents  as  they  are  pro- 
duced. 

For  the  convenience  of  the  reader,  the  discussion  which  follows  is  organized 
along  the  same  lines  as  the  guidelines  given  in  Section  2.  In  fact,  the  sub- 
section numbers  are  the  same,  and  we  have  repeated  some  information,  viz.  le- 
vel definitions  and  document  types,  to  avoid  repeated  referencing  to  Section 
2.1.  We  have  also  included  a summary  of  the  document  definitions  given  in 
Section  2.2;  these  are  given  in  italics  below. 

3. 1 Assessment  of  Level  I Documentation 


As  stated  earlier,  this  phase  deals  with  the  mechanics  of  operation  of  the 
computer-coded  version  of  the  model  on  a particular  computer.  The  relevant 
document  types  are: 
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(1-1)  Operations  Manual, 

(1-2)  Data  Base  Description:  Physical/Logical  Characteristics,  and 

(1-3)  Software  Description:  User  Level. 

Each  of  these  will  be  addressed  in  turn  below. 

3.1.1  The  Operations  Manual 

The  purpose  of  this  manual  is  to  provide  computer  operations  personnel  with  a 
description  of  the  software  end  of  the  operational  environments  so  that  the 
software  can  be  run . 

The  document  "Operations  Guide:  Midterm  Oil  and  Gas  Supply  Model"  [C-23] 

(dated  2/79)  has  as  its  stated  purpose  to  provide  "the  instructions  and  refer- 
ence information  necessary  to  use  the  DOE  Midterm  Oil  and  Gas  Supply  Model  at 
a site  other  than  the  DOE/EIA  Facility."  This  purpose  was  not  achieved.  We 
have  been  able  to  complete  a computer  run  of  the  model,  but  only  after  repeat- 
ed sessions  with  DOE  staff;  it  would  not  have  been  possible  to  get  that  far 
without  such  close  and  repeated  assistance.  One  of  the  major  deficiencies  in 
this  draft  version  (2/79)  of  the  Guide  is  that  the  two  most  important  Appendi- 
ces, B and  D,  are  not  included  in  the  document.  Appendix  B was  meant  to  pro- 
vide the  information  needed  to  read  the  tape  files  into  the  computer,  and 
Appendix  D was  meant  to  include  sample  outputs.  We  understand  that  the  appen- 
dices were  not  included  due  to  their  great  bulk.  We  feel  that  it  is  incumbent 
upon  authors  of  such  a 

^Lettered  references  can  be  found  in  Appendix  C. 
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document  to  discover  ways  to  abridge  such  information  so  that  it  can  be  in- 
cluded. Otherwise  a document  is  incomplete  and  must  remain  so. 

There  is  a disturbing  assumption  in  the  Guide  that  certain  structures  and 
software  systems  are  "standard”  across  IBM  machines.  This  should  be  verified 
(or  noted  to  be  true),  since  the  existence  of  IBM's  utility  software  IEHMOVE, 
IEBPTPCH,  and  SORTD  as  fixed,  unchanged  routines  across  similar  IBM  equipment 
is  a prerequisite  for  this  phase  of  model  reproduction.  We  are  also  concerned 
about  the  assumption  that  "the  overlay  options  of  IEWL"  and  the  resultant 
overlay  structure  used  in  the  model's  software  are  the  same  across  IBM  instal- 
lations. Again,  if  this  is  true,  some  statement  to  that  effect  should  be  in- 
cluded. More  importantly,  if  it  is  untrue,  the  notion  of  providing  access  to 
the  model  via  this  document  is  an  incorrect  one,  unless  alternatives  are  pre- 
sented. 

We  understand  that  this  manual  has  recently  been  revised  to  include  complete 
listings  of  the  Job  Control  Stream  needed  to  complete  a computer  run,  as  well 
as  sample  problems  sufficient  to  make  clear  the  format  of  both  input  and  out- 
put and  their  interpretation.  Statements  about  core  storage,  portability  of 
IBM  utility  packages,  tape-drive  requirements  and  other  system-related  fea- 
tures of  the  code  are  said  to  have  been  added.  We  have  not  reviewed  this  re- 
vised document  because  a copy  of  it  was  received  after  our  cutoff  date  for  in- 
clusion in  the  documentation  assessment. 

3.1.2  Data  Base  Description:  Physical/Logical  Characteristics 


In  [1]  this  document  was  called  the  Data  Base  Specification . Its  purpose  is 
to  specify  the  identification , logical  characteristics , and  physical  charac- 
teristics of  a particular  data  base. 
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As  of  the  time  of  this  documentation  assessment  (2/79),  there  were  no  docu- 
ments provided  that  satisfied  these  documentation  needs.  The  "Systems  Instal- 
lation and  Operations  Guide”  [C-7]  contains  minimal  information  on  data  set 
names  and  sizes  but  no  information  on  meaning,  logical  characteristics,  data 
entry,  update  methods,  or  references.  The  "Medium-Run  Oil  and  Gas  Supply  Mod- 
el, 1977  Data  Update"  [ C— 1 1 ] (hereafter  referred  to  as  "1977  Data  Update")  is 
also  inadequate  to  this  task.  It  contains  no  specific  data  base  information 
of  the  type  described  above. 

3.1.3  Software  Description:  User  Level 

The  purpose  of  this  document  is  to  describe  sufficiently  the  functions  per- 
formed by  the  software  so  that  the  user  organization  can  determine  the  logis- 
tics of  its  applicability  and  how  to  put  it  into  operation.  It  should  serve 
as  a reference  document  for  preparation  of  input  data  and  parameters  and  for 
interpretation  of  results . 

There  was  nothing  in  the  materials  furnished  to  us  that  approximated  this 
document. 

3.1.4  Degree  of  Attainment  of  Phase  I Objectives 

The  delivered  document  materials  do  not  satisfy  any  phase  I requirements. 

3. 2 Assessment  of  Level  II  Documentation 

The  second  level  identified  in  Section  2 is  the  one  in  which  one  tries  to  use 
the  code  to  implement  the  model.  In  this  situation,  one  needs  sufficient 
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information  so  as  to  be  able  to  run  the  model  with  new  input  data.  The  rele- 
vant documents  are  those  specified  in  Level  I above  and: 

( I I— 1 ) Mathematical  Formulation  Description; 

( I 1—2 ) Data  Requirements  Report:  Sources,  Transformations  and 

Justification;  and 
( I 1—3 ) Process  Description. 

The  Midterm  Oil  and  Gas  Supply  Model  reports  will  now  be  evaluated  in  light  of 
these  document  types. 

3.2.1  Mathematical  Description 

This  report  describes  the  complete  details  of  the  mathematical /logical  model 
including  assumptions  and  hypotheses , the  rationale  for  its  form , and  the  re- 
strictions on  its  use. 

There  is  no  of  documentation  for  the  Midterm  Oil  and  Gas  Supply  Model  that 
serves  the  function  of  the  Mathematical  Description.  Some  examples  of  the 
omissions  in  the  existing  documentation  will  be  presented  below,  in  order  to 
illustrate  the  need  for  a more  complete  and  better-organized  approach  to  the 
documentation  of  this  model.  Each  item  in  the  definition  of  the  ideal  above 
is  addressed  separately  below. 

3.2. 1.1  Mathematical/Logical  Structure  of  the  Model,  Including  Assumptions 
and  Hypotheses 

There  are  four  separate  documents  [C-10,  11,  12,  22]  which  present  pieces  of 
the  mathematical/logical  structure  of  the  Midterm  Oil  and  Gas  Supply  Model,  as 
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well  as  a number  of  ancillary  documents  [C-14,  15,  16,  17,  18,  19,  20,  21] 
which  contain  some  more  detailed  descriptions.  In  order  to  understand  the 
model’s  representation  of  the  underlying  process  of  oil  and  gas  supply,  all  of 
these  documents  must  be  read.  Even  though  approximately  twenty  documents  de- 
scribing the  Midterm  Oil  and  Gas  Supply  Model  have  been  studied,  some  calcula- 
tions performed  in  the  model  remain  unexplained.  For  example,  the  "Oil  and 
Gas  Midterm  Supply  Model:  Methodology  Description"  [ C— 1 2 ] (hereafter  referred 

to  as  the  "Methodology  Document")  describes  a number  of  cash  flow  categories 
such  as  investments  (expensed  and  capitalized),  production  costs  and  annual 
cash  expenses  (depreciation,  depletion)  that  must  be  used  in  determining  the 
cost  of  producing  various  quantities  of  oil  and  gas.  Knowing  how  these  data 
items  are  used  is  crucial  to  an  understanding  of  the  model,  but  this  usage  is 
unexplained. 

One  report,  "U.  S.  Oil  and  Gas  Supply  Computer  Program  Documentation"  [ C— 2 2 ] , 
dated  1973,  has  been  cited  to  us  as  the  documentation  of  the  mathematical 
structure  of  the  model.  The  only  legible  section  of  this  document,  Section 
VI,  was  added  at  some  later  time  by  staff  of  ICF,  Inc.,  the  contractor  who 
prepared  the  model,  to  detail  the  changes  made  to  the  original  model. 
Unfortunately,  that  section  does  not  explain  which  pieces  of  the  original 
model  are  replaced  or  altered. 

Section  VI  also  presents  a listing  of  equations  and  brief  descriptions  of 
variables.  However,  reading  a lengthy  list  of  equations,  without  explanation 
or  justification,  is  not  much  easier  than  reading  computer  code,  nor  more 
enlightening.  For  our  assessment  efforts,  we  need  additional  elaboration  of 
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the  reasoning  behind  the  mathematics  of  each  section  of  the  model.  A useful 
guideline  could  be  the  comments  related  to  equations  98-107,  found  in  Section 
VI  (page  5.20)  of  that  report. 

A description  of  the  model  should  not  only  include  the  mathematical  statement 
of  the  model  but  should  discuss  each  of  the  assumptions  made.  This  discussion 
should  include  for  each  assumption,  an  explanation  of  its  effect  on  model  out- 
put values.  In  the  case  of  the  Midterm  Oil  and  Gas  Supply  Model,  assumptions 
are  scattered  throughout  various  documents.  They  are  nowhere  collected  or 
classified,  and  no  indication  is  given  as  to  how  they  individually  and  jointly 
limit  the  applicability  of  the  model.  The  model  can  and  should  be  used  to 
study  the  effects  or  implications  of  some  assumptions,  and  the  results  of  such 
a study  should  be  reported  to  policy  makers. 

This  lack  in  the  model  documentation  is  especially  serious  in  a policy  making 
environment  where  the  decision  makers  are  not  the  analysts.  Without  inform- 
ation about  the  assumptions  of  the  model  and  any  competing  models,  it  is  im- 
possible for  a decision  maker  to  choose  among  alternative  models  or  even  to 
have  confidence  in  the  output  of  a specific  model. 

Although  many  assumptions  are  articulated  in  the  Midterm  Oil  and  Gas  Supply 
Model  documentation,  others  are  left  unstated  or  their  implications  are  not 
explained.  For  example,  decline  rates  are  assumed  to  be  constant.  The  impli- 
cation of  a constant  decline  rate  is  that  regardless  of  future  price 
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increases,  there  will  be  no  extraordinarily  intensified  extraction  from  old 
fields  which  could  affect  the  quantity  of  oil  produced.  This  potential  effect 
should  be  discussed  further  in  the  report. 

We  recommend  that  a list  of  assumptions  be  compiled  and  checked  for  complete- 
ness by  the  model  analysts  and  computer  programmers  responsible  for  the  final 
model  and  computer  code. 

3. 2. 1.2  Theoretical  and  Analytic  Rationale  for  the  Mathematical/Logical  Form 
Including  Some  Discussion  of  Alternatives 


None  of  the  documents  discuss  alternative  approaches,  nor  are  discussions  pre- 
sented for  any  of  the  equations  used  in  the  model.  We  present  one  example  to 
illustrate  why  we  believe  each  of  the  equations  should  be  discussed  in  more 
detail. 

Hubbert's  factors  are  used  to  calculate  growth  of  the  resource  base  from  "ex- 
tensions and  revisions,"  but  no  justification  is  presented  as  to  why  these 
factors  are  appropriate  or  why  national  figures  can  be  applied  to  all  regions. 
Sensitivity  of  the  output  to  these  factors  is  not  reported.  Furthermore,  no 
reference  to  the  original  work  of  Hubbert  is  provided,  thereby  making  the  un- 
initiated assessor’s  job  of  evaluating  these  formulas  much  more  difficult. 
Similar  criticisms  apply  for  each  equation  used  in  the  model. 
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In  addition  to  providing  the  rationale  for  the  form  of  each  piece  of  the  mod- 
el, some  documentation  describing  the  choice  of  a general  engineering-process 
approach  as  opposed,  for  instance,  to  an  econometric  approach  should  also  be 
presented.  This  discussion  should  include  brief  narratives  that  describe  in 
general  terms  the  alternative  models  available  and  the  strengths  and  limita- 
tions of  each.  This  should  not  be  a relatively  expensive  and  time-consuming 
effort.  Any  good  research  or  development  project  should  begin  with  a litera- 
ture search  to  discover  what  has  already  been  done.  This  should  be  summarized 
for  the  record. 

3.2. 1.3  Restrictions  on  the  Use  of  the  Model 


The  documentation  of  the  Oil  and  Gas  Model  contains  no  explicit  statement  of 
the  time  horizon  for  forecasting.  We  consider  this  a serious  omission.  The 
abstract  of  the  Methodology  Document  states  "the  Oil  and  Gas  Supply  Model  is  a 
computer-based  model  which  projects  domestic  oil  and  natural  gas  production 
for  15  years  based  on  economic  and  engineering  factors  which  affect  oil  and 
gas  supply."  One  question  which  naturally  arises  is  whether  this  model  can  be 
used  to  predict  gas  supply  in  either  the  short  term  (two  or  three  years  from 
the  present),  or  the  longer  term  (twenty-five  to  thirty  years),  or  if  the  un- 
derlying methodology  requires  that  model  outputs  be  considered  valid  only 
after  some  adjustment  period.  Statements  describing  modes  of  applications  of 
this  model  and  its  limitations  would  be  useful. 

Since  the  Model  Formulation  Description  is  intended  to  serve  as  the  primary 
source  of  information  about  a model,  we  recommend  the  preparation  for  the  Mid- 
term Oil  and  Gas  Supply  Model  of  such  a document  along  the  lines  indicated 
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above.  This  document  should  include  a description  of  each  of  the  items  listed 
in  the  ideal  above  and  should  include  references  to  any  other  documents  needed 
to  understand,  use,  validate,  or  alter  the  model. 

3.2.2  Data  Requirements  Report:  Sources,  Transformations  and  Justification 

For  all  input  variables  and  "hardivired"  parameters , this  report  describes  data 
sources , transformations  and  justifications  thereof , data  validation  proce- 
dures and  acceptable  data  ranges . 

The  1977  Data  Update  [C-ll]  supplies  some  of  the  information  described  above, 
but  omits  some  data  values  and  is  incomplete  in  its  discussion  of  others.  For 
example,  according  to  the  Methodology  Document,  in  order  to  calculate  the  seg- 
ment of  the  resource  base  which  is  economic  to  pursue,  "a  special  run  of  the 
engineering  and  minimum  acceptable  price  sectors  of  the  model  are  executed 
with  a hypothetical,  ambitious  drilling  program."  No  definition  or  explana- 
tion of  a "hypothetical,  ambitious  drilling  program"  is  provided,  nor  is  the 
purpose  of  executing  this  run  explained.  Similarly,  although  there  is  a 
lengthy  discussion  in  the  Data  Update  on  how  to  allocate  drilling  among  old 
and  new  rigs,  no  data  is  provided  for  the  inital  rig  and  plant  capacity. 

The  1977  Data  Update  does  not  give  sources  for  all  of  the  data;  e.g.,  the  off- 
shore escalation  factor  is  given  as  2 percent,  with  no  justification  or  refer- 
ence. While  the  report  includes  a detailed,  yet  incomplete,  discussion  of  the 
processes  for  transforming  data  into  the  finding  rates  needed  by  the  model,  it 
is  less  specific  in  describing  other  necessary  transformations.  For  example, 
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calculation  of  the  secondary  and  tertiary  recovery  and  intensification  factors 
involves  some  unexplained  application  of  Hubbert  factors,  utilizes  referenced 
but  undefined  tertiary  recovery  production  forecasts,  and  applies  some  unspec- 
ified adjustment  factors. 

Numerical,  statistical,  and  forecasting  techniques  used  for  parameter  estima- 
tion are  not  described.  For  example,  in  describing  the  calculation  of  finding 
rates  the  report  states  that  regression  analyses  were  used  and  includes  tables 
of  the  results  of  these  analyses.  The  document  does  not  specify,  however,  the 
method  of  regression,  the  regression  code  used,  or  the  measures  of  signifi- 
cance associated  with  the  parameter  estimates.  Finally,  there  is  no  descrip- 
tion of  any  data  validation  efforts  nor  is  there  any  attempt  to  identify 
reasonable  ranges  on  the  data.  Although  the  report  does  characterize  by 
"high,  medium,  or  low"  the  sensitivity  of  the  model  to  the  data  items  on 
several  lists,  this  information  is  of  little  value  for  assessment  purposes 
without  an  accompanying  description  of  how  these  sensitivities  were  deter- 
mined, including  numerical  ranges  for  the  data  items,  identification  of  which 
output  items  were  sensitive  to  changes,  and  quantitative  definitions  for  the 
measures  of  sensitivity. 

A data  requirements  description  is  an  essential  document,  and  it  is  strongly 
recommended  that  DOE  initiate  efforts  to  produce  one.  The  1977  Data  Update 
would  supply  a framework  and  some  of  the  necessary  information,  but  it  needs 
much  improvement  and  polishing  before  it  will  satisfy  the  data  information 
needs  of  users  and  assessors. 


-31- 


3.2.3  Process  Description 

This  report  provides  a comprehensive  description  of  the  underlying  physical, 
economic,  technological , and  behavioral  processes  to  be  modeled. 

In  the  case  of  the  Midterm  Oil  and  Gas  Supply  Model,  the  purpose  of  the  pro- 
cess description  is  to  provide  an  introductory,  but  thorough,  description  of 
engineering,  economic,  and  political  factors  that  are  believed  to  affect  the 
future  national  supply  of  oil  and  gas. 

There  is  no  such  document  for  the  model  and  although  the  Methodology  Document 
includes  a section  on  the  supply  process,  it  does  not  describe  adequately  all 
aspects  of  the  process,  nor  identify  an  existing  source  for  this  information. 
Central  topics  omitted  from  the  discussion  in  the  Methodology  Document  in- 
clude: the  effects  of  regulation  and  taxation  on  oil  and  gas  supply;  the 

present  configuration  of,  and  the  barriers  to  entry  into,  the  industry;  the 
differences  between  the  geological  estimates  and  resource  base;  and  the  manner 
in  which  mineral  property  rights  are  acquired  through  leasing  and  royalty 
agreements. 

In  contrast,  three  reports  "A  Comparative  State-of-the-Art  of  Assessment  of 
Gas  Supply  Modeling"  [C-3],  "A  Comparative  State-of-the-Art  of  Assessment  of 
Oil  Supply  Modeling”  [C— 4 ] , and  "Oil  and  Gas  Resources — Welcome  to  Uncertain- 
ty" [C-23]  provide  a more  complete  description  of  the  underlying  process  being 
modeled.  If  these  reports  accurately  portray  the  oil  and  gas  supply  process, 
they  should  be  referenced  as  sources  for  such  process  information.  If  not, 

DOE  should  consider  producing  such  a report,  comparable  in  depth  but  rectify- 


ing existing  deficiencies 
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3. 2. A Degree  of  Attainment  of  Phase  II  Objectives 

The  delivered  documentation  does  not  furnish  any  capability  for  experimenta- 
tion beyond  rote  (Phase  1)  operation. 

3.3  Assessment  of  Level  III  Documentation 


The  third  level  of  model  use  identified  in  Section  2 is  the  model  maintenance 
phase.  During  this  phase  one  attempts  to  make  changes  to  the  existing  model. 
The  two  suggested  reports  are: 

(III-l)  Software  Description:  Programmer  Level;  and 

(III-2)  Maintenance  Log. 

These  will  now  be  discussed. 

3.3.1  Software  Description:  Programmer  Level 

The  purpose  of  this  document  is  to  provide  the  maintenance  programmer  with  the 
information  necessary  to  understand  the  code  architecture,  the  logic  of  each 
of  the  subprograms,  and  the  operating  environment  to  a sufficient  depth  so 
that  the  programmer  is  capable  of  maintaining,  correcting , and  enhancing  the 
computer  code. 

This  report  is  an  important  information  source  to  anyone  concerned  with  as- 
sessment, as  well  as  to  those  who  must  operate  and  run  the  model.  Whereas  the 
"Software  Description:  User  Level”  (see  sect.  2.2.3)  provides  an  overview  and 
instructions  on  how  to  use  the  computer  realization  of  the  model,  the  program- 
mer level  manual  gives  details  of  the  programs  themselves. 
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The  report  "U.  S.  Oil  and  Gas  Supply  Computer  Program  Documentation"  [ C— 1 4 ] 
can  serve  as  an  example  but,  unfortunately,  not  as  an  operational  embodiment 
of  the  programmer-level  document.  It  is  dated  1973  and  contains  no  reference 
to  subsequent  updates.  There  is  no  obvious  one-to-one  correspondence  between 
the  subroutines,  data  input,  and  output  reports  described  in  the  document  and 
those  of  the  computer  software  we  received.  Furthermore,  the  available  copies 
of  this  report  are  largely  illegible. 

3.3.2  Maintenance  Log 

This  log  identifies  and  records  changes  made  to  the  model  and/or  its  data . 

For  the  assessment  team,  this  log  would  be  most  useful  in  that  it  would  pro- 
vide a chronological  description  of  model  evolution.  Although  it  is  most  dif- 
ficult to  make  log  entries  during  a crisis,  procedures  exist  according  to 
which  it  is  done  routinely,  e.g.,  by  air  traffic  controllers  in  saturation 
traffic,  and  for  various  analogous  activities.  Log  maintenance  is  especially 
important  in  "crash  mode,"  that  is,  in  periods  of  tense  activity  and  rapid 
changes  (when  working  against  deadlines)  to  guard  against  chaos  "when  the 
shooting  dies  down."  There  is  no  documentation  of  this  type  in  any  of  the 
reports  listed  in  Appendix  C. 

3.3.3  Degree  of  Attainment  of  Phase  III  Objectives 

The  documentation  provides  no  assistance  for  modification  of  the  model. 
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3.4  Assessment  of  Level  IV  Documentation 


This  level  of  documentation  contains  information  needed  for  conducting  a 
third-party  (or  arm's  length)  model  assessment.  The  documents  included  in 
this  section  address  topics  relevant  to  the  assessment  effort  such  as  reports 
of  efforts  to  apply,  verify,  or  assess  the  model.  It  is  also  the  catch-all 
for  any  other  reports  relating  to  the  model,  since  all  model  documentation 
should  be  reviewed  by  the  assessor.  Typical  reports  to  be  found  in  this  broad 
category  include: 

(1)  Assessment  Report,* 

(2)  Historical  Record, 

(3)  Model  Suomary, 

(4)  Model  Application  Report,  and 

(5)  any  of  the  documents  in  [1,2,5]  or  any  other  documents  written  about 
the  model  but  not  specifically  listed  above. 

The  first  four  documents  in  this  category  are  discussed  below. 

3.4.1  Assessment  Report 


Documentation  of  model  assessment  activities  should  include  the  results  of  any 
tests  of  the  model's  output  in  terms  of  comparisons  to  historical  data , ac- 
ceptability by  the  user , statistical  measures,  comparative  results  of  alterna- 
tive formulations . 


*Note  that  this  document  refers  to  assessment  in  which  the  model  developer 
participates,  at  least  in  the  planning  stage. 
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In  the  case  of  the  Midterm  Oil  and  Gas  Supply  Model,  documentation  in  this 
area  is  apparently  nonexistent.  Two  prof erred  documents,  however — "FEA  Model 
of  Oil  and  Gas  Supply:  Data  Validation  and  Update  " [C-14],  and  the  1977  Data 

Update — list  three  levels  of  sensitivity  (high,  medium,  low)  of  model  outputs 
to  variations  in  values  of  selected  input  parameters.  If  these  were  arrived 
at  judgmentally , the  documentation  should  so  state.  If  these  estimates  re- 
sulted from  some  testing,  the  tests  should  be  described.  Similarly,  accep- 
tance testing  of  the  computer  code  and  statistical  tests  of  the  regressions 
may  have  been  performed,  but  documentation  describing  these  test  efforts  has 
not  been  provided.  We  recommend  documentation  of  all  validation  procedures 
performed  on  this  model  and  of  the  test  results  which  can  be  reconstructed  at 
this  time. 

Obviously,  if  proper  testing  is  performed  at  appropriate  stages  of  a model's 
life  cycle  and  if  this  testing  is  well-documented,  an  assessor's  job  is 
easier,  and  the  assessment  is  more  likely  to  be  cogent  and  complete.  Without 
such  documentation,  the  assessment  team  must  recheck  much  that  may  already 
have  been  checked.  Extending  this  notion,  if  the  model  developers  performed 
all  the  validation  and  verification  tests  possible,  and  provided  documentation 
of  those  efforts  along  with  the  other  documents  requested  in  this  report,  as- 
sessment efforts  could  be  limited  to  checking  for  reproducibility. 
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3.4.2  Historical  Record 


This  report  describes  the  questions  or  problems  which  led  to  the  decision  that 
the  model  was  needed,  how  the  model  is  to  be  used  to  address  these  issues , and 
the  advantages  and  disadvantages  of  alternatives. 

None  of  the  documents  listed  in  Appendix  C contain  historical  information  of 
the  type  described  herein  for  the  DOE  Midterm  Oil  and  Gas  Supply  Model.  Ab- 
sent documented  input  from  the  policy  makers,  this  information  is  best  known 
to  the  model  builders  and  should,  therefore,  be  constructed  by  them. 

3.4.3  Model  Summary 


This  report  provides  a concise  summary  of  the  model  so  that  other  users  and 
analysts  can  determine  if  it  is  of  interest  to  them. 

The  document  entitled  "Description  of  Method  Used  to  Forecast  Domestic  Oil  and 
Gas  Supply”  [C-4]  satisfies  this  need.  It  is  a concise  overview  of  the  model 
and  should  become  part  of  the  formal  documentation.  We  recommend  that  a ref- 
erence be  provided  to  the  work  of  Hubbert  and  that  definitions  be  provided  in 
this  summary  for  the  following  technical  terms:  resource  base,  new  pays,  in- 

ferred and  indicated  resources,  and  recovery  methods. 

3.4.4  Model  Application  Report 


This  report,  directed  toward  those  who  would  use  the  model  results  to  make  de- 
cisions, should  present  results  of  exercising  the  model  to  obtain  answers  to 
specific  questions  posed  by  executives  and  to  study  the  behavior  of  the  prob- 
lem environment  as  it  is  represented  by  the  model. 
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The  "Annual  Report  to  Congress,"  [C-l ] is  similar  to  the  Model  Application  Re- 
port described  above.  This  document,  however,  addresses  the  entire  set  of 
models  including  PIES  and  the  various  models  which  supply  input  to  PIES  and/- 
or process  output  from  PIES.  It  is  therefore  diffcult  to  locate  and  identify 
within  the  report  those  statements  which  apply  exclusively  to  the  Midterm  Oil 
and  Gas  Supply  Model.  The  sections  entitled  "Summary,"  "Introduction,"  and 
"Energy  System  Projections,"  and  the  "Oil”  and  "Gas"  chapters  of  "The  Sources 
of  Energy"  all  contain  statements  related  to  oil  and  gas  supply,  but  it  is  not 
possible  to  isolate  from  the  context  the  role  of  MOGSM  in  the  generations  of 
these  forecasts. 

In  describing  data  values,  the  Annual  Report  fails  to  distinguish  among  model 
inputs  (and  assumptions),  model  outputs,  and  external  reference  data  (and  as- 
sumptions). For  example,  that  report  states  that  domestic  production  of  pe- 
troleum liquids  is  forecast  to  decline  by  about  9 percent  from  1977  to  1985 
(page  138)  and  indicates  that  the  accuracy  associated  with  this  estimation  is 
"still  quite  high;  confidence  in  the  forecast  estimate  stems  primarily  from 
the  accuracy  associated  with  the  portions  of  the  resource  base  that  contribute 
most  of  the  reserve  additions,  primarily  old  fields."  However,  what  is  not 
stated  is  that  the  decline  is  therefore  the  result  of  an  assumption — namely, 
that  decline  rates  are  constant  and  that  the  decline  rate  is  an  input  value 
which  DOE  has  chosen  to  be  the  current  rate  of  production  used  by  the  oil  in- 
dustry. 

*Since  the  start  of  this  study,  the  name  has  been  changed  from  PIES  to  MEMM  or 
Midrange  Energy  Market  Model. 
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Because  the  Annual  Report  discusses  assumptions  throughout  its  text,  and  col- 
lects some  but  not  all  of  these  into  paragraphs  labeled  "Assumptions,"  it 
would  be  difficult  to  determine  if  all  model  assumptions  are  omitted.  For  ex- 
ample, in  the  1977  Data  Update,  the  rate  of  return  on  investment  is  defined  in 
a single  short  sentence  as  8 percent,  with  no  source  or  justification  and  no 
ranges  on  this  value,  which  apparently  is  taken  to  remain  constant  over  all 
years,  regions,  recovery  methods,  etc.  That  same  report  identifies  the  rate 
of  return  as  an  input  parameter  to  which  model  sensitivity  is  "Hi."  Yet 
neither  this  sensitivity,  nor  the  particular  value  (8  percent),  is  mentioned 
in  the  previously  referenced  sections  of  the  Annual  Report. 

3.4.5  Degree  of  Attainment  of  Phase  IV  Objectives 

While  "model  assessment"  is  not  at  present  completely  defined,  it  is  clear 
that  signficant  components  of  an  assessment  process — operation  of  the  computer 
programs,  comparison  of  the  rationale  of  the  model's  structure  and  model  out- 
puts to  the  model  objectives,  determination  of  limitations  in  range  and 
scope — cannot  be  accomplished  by  using  the  computer  programs  and  the  documen- 
tation alone.  Direct  assistance  by  the  developers  is  necessary. 
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4.  RECOMMENDATIONS  AND  CONCLUSIONS 


Our  assessment  of  the  documentation  of  the  Midterm  Oil  and  Gas  Supply  model 
has  led  to  several  conclusions  and  general  recommendations.  First,  on  the 
subject  of  documentation  assessment,  existing  guidelines  for  evaluation  of 
models  and  for  evaluation  of  supporting  documentation  do  not  constitute  a 
clear  application  of  a highly  developed  theory;  they  are  merely  common  sense 
clothed  in  formal  language.  In  this  absence  of  a definitive  set  of  rules  and 
criteria,  we  recommend  that  others  undertaking  a similar  effort  follow  our  ap- 
proach. That  is,  we  decided  on  a general  description  of  the  documentation 
necessary  for  assessment,  and  then  compared  the  available  documents  to  this 
description.  This  recourse  to  a standard,  even  one  possibly  subject  to  even- 
tual drastic  revision,  is  the  first  step  in  transforming  the  assessment  of 
model  documentation  from  the  realm  of  literary  criticism  to  that  of  a measure- 
ment process.  Second,  also  concerning  assessment,  we  strongly  recommend  that 
at  least  one  member  of  the  team  assessing  documentation  should  be  unfamiliar 
with  the  model,  thereby  relying  on  the  documents  as  the  only  source  of  inform- 
ation about  it.  Persons  already  familiar  with  the  model  may  take  for  granted 
some  piece  of  information  and  thus  fail  to  recognize  its  omission  from  the 
documentation  or  any  lack  of  clarity  in  the  exposition. 

On  the  subject  of  model  assessment:  after  considerable  effort,  we  have  been 

able  to  obtain  an  overview  of  the  Midterm  Oil  and  Gas  Supply  Model,  sufficient 
for  a partial  model  assessment.  This  is  not  to  say  that  the  documentation  is 
sufficient.  Indeed,  much  labor  and  information  acquired  outside  the  documents 
were  required  to  reach  our  current  state  of  model  understanding,  beset  with 
gaps  as  it  still  is.  A fuller  understanding  of  the  model  would  require 
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reading  every  line  of  computer  code  and  having  the  modelers  explain  any  dis- 
crepancies between  the  mathematical  formulation  and  computer  implementation. 
Unless  better  documentation  is  developed,  others  desiring  to  use  or  to  assess 
the  model  will  have  to  repeat  our  efforts  only  to  reach  the  same  level  of  in- 
complete understanding.  In  essence,  this  means  that  with  much  effort  even  a 
poorly  documented  model  can  be  assessed  if  (1)  those  aspects  of  the  model  for 
which  additional  information  is  essential  can  be  identified  and  (2)  this  addi- 
tional information  can  be  obtained  directly  from  persons  intimately  familiar 
with  the  model  (e.g. , DOE  staff  or  contractor). 

However,  since  we  do  not  intend  by  our  own  efforts  to  add  to  the  body  of  docu- 
mentation of  the  Midterm  Oil  and  Gas  Supply  Model,  we  must  address  the  problem 
of  recommending  ways  to  improve  access  to  it  through  the  p oet  hoc  preparation 
of  documentation.  This  situation  is  different  from  advancing  methods  and 
techniques  for  the  a priori  development  of  a documentation  plan  for  use  in  a 
modeling  project.  Now,  we  must  accept  that  for  some  time  to  come,  models  (for 
example  this  model)  will  be  developed,  delivered,  and  used  with  documentation 
that  does  not  satisfy  any  of  the  guidelines  under  development.  In  Section  3, 
we  discussed  each  recommended  document  type  and  evaluated  this  model's  docu- 
mentation in  light  of  that  set.  We  also  indicated  documentation  which  seems 
to  be  crucial  to  the  process  of  model  assessment,  and  should  therefore  be  pre- 
pared after  the  fact  if  it  has  not  been  done  during  model  development.  This 
"post  hoc"  set  is  collected  below: 
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For  Phase  I usage — 

(1)  Operations  Manual, 

(2)  Data  Base  Description:  Physical/Logical  Characteristics, 

(3)  Software  Description:  User  Level. 

For  Phase  II  usage  by  "sophisticated"  users — 

(4)  Data  Requirements  Report. 

For  Phase  III  usage — 

(5)  Software  Description:  Programmer  Level 

(6)  Mathematical  Description, 

(7)  Maintenance  Log  (to  the  extent  that  the  pertinent  information  can 

be  assembled  now) 

To  satisfy  the  needs  of  a broad  spectrum  of  users  the  Phase  II  post  hoc  docu- 
ment should  be  augmented  by  (8)  Process  Description.  This  document  is  needed 
in  any  case  for  model  assessment  (Phase  IV). 

We  have  stated  earlier  the  crucial  importance  of  good  documentation  in  provid- 
ing the  means  for  using  models  and  the  professional  requirement  for 
enabling  replication  of  the  results  of  analysis.  There  are  other  benefits 
to  be  gained  from  a thorough  documentation  effort,  including: 

(a)  training  users  inside  the  sponsoring  organization 

(b)  introducing  the  model  readily  to  interested  parties,  outside  the 
sponsoring  organization,  and 

(c)  discovery  of  model  enhancements  when  modelers  structure  their 
knowledge  of  the  model  in  order  to  communicate  it  to  others. 
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Proper  documentation  cannot  be  taken  for  granted.  In  the  introduction,  we 
have  noted  reasons  related  to  deadline  pressure  and  reluctance  to  "freeze" 
models  that  account  for  scanting  of  documentation  by  model  developers.  This 
situation  is  likely  to  endure  indefinitely  because  of  the  critical  nature  of 
the  problems  addressed  through  large  scale  models  and  the  expected  deficien- 
cies in  background  data.  Therefore,  model  sponsors  should  take  actions  de- 
signed to  assure  adequate  documentation.  These  include  at  least,  specifica- 
tion of  information  requirements  in  negotiating  contracts  for  model  develop- 
ment and  provision  of  funding  and  time  schedules  for  documentation  as  line 
items  in  such  contracts.  Also,  in  contract  awards,  the  same  kind  of  scanting 
of  qualifications  and  specification  of  level  of  participation  should  be  given 
to  proposed  documentors  as  is  now  customarily  given  to  analysts. 

We  believe  that  the  guidelines  described  in  this  report  should  be  provision- 
ally adopted  for  modeling  activity  under  DOE  aegis  because  the  information 
content  described  represents  substantial  consensus  among  analysts  and  is  not 
likely  to  be  revised  in  a way  that  invalidates  documentation  thus  produced. 
Moreover,  we  assert  that  information  designated  in  the  guidelines  is  essential 
in  that  no  revisions  could  result  in  substantial  excess  effort  having  been  ex- 
pended. 

In  invoking  consensus,  we  remark  that  although  we  relied  on  [5]  and  [2]  as 
background,  noting  that  the  intention  in  [2]  was  exhaustive  documentation  for 
planning,  usage,  evaluation,  and  archive  purposes,  we  proceeded  independently 
to  develop  these  guidelines  from  first  principles,  as  it  were  (through  inter- 
views, reading,  and  discussion  to  identify  information  requirements  for  a 
spectrum  of  application  and  then  to  assemble  them  according  to  a rational 
plan). 
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The  result  differs  from  the  material  in  [2]  primarily  in  defining  a more  con- 
cise set  of  documents  and  having  discarded  a set  of  criteria  which  we  regarded 
as  too  highly  precise  and  structured  at  the  current  stage  of  elaboration  of 
documenting  methodology.  The  guidelines  are  substantially  similar  to  those  of 
[5]  except  that  they  expand  the  document  set  (but  not  the  total  information 
requirements)  according  to  specific  levels  of  model  usage.  A statement  that 
we  believe  that  the  guidelines  should  be  adopted  but  that  they  are  subject  to 
some  change  and  that  they  should  not  be  rigidly  applied,  requires  some  clarif- 
ication. We  believe  that  (at  this  time,  at  any  rate)  model  documentation 
giudelines  cannot  be  stated  with  the  force  of,  say,  specifications  for  mili- 
tary equipment.  We  intend  that  the  guidelines  should  become  a rational  basis- 
for  a priori  agreements  between  sponsors  and  developers  as  to  the  documenta- 
tion that  shall  accompany  models  (and  according  to  which  mutually  agreeable 
criteria  for  judging  that  documentation  can  be  derived). 

Finally,  we  offer  two  conclusions  of  a methodological  nature.  We  believe  that 
production  of  documentation  pari  passu  with  the  development  of  a model  will 
maximize  efficiency  and  quality  of  documentation  even  for  those  portions  of 
the  material  which  would  require  modification  as  the  model  is  refined.  Pos- 
sible omission  of  relevent  material  will  be  minimized,  milestones  will  be  more 
easily  scheduled,  and  no  time  losses  should  be  incurred  in  reworking  material- 
-it  is  as  unlikely  that  a narrative  will  be  "right"  without  revision  as  that  a 
computer  program  can  be  written  linearly  without  bugs. 
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As  in  the  case  of  document  asessment,  some  member  of  the  document  producing 
team,  preferably  an  editor,  should  be  unfamiliar  with  the  details  of  the  model 
or  its  methodology,  and  therefore  be  in  a good  circumstance  to  detect  depar- 
tures from  clarity  in  the  exposition. 

A belief  that  has  guided  our  efforts  in  this  project  is  that  if  we  can  get 
some  notion  of  what  constitutes  "good"  (minimal,  maximal,  or  whatever)  docu- 
mentation and  if  this  can  be  cast  into  a set  of  guidelines  in  such  a way  as  to 
avoid  bureaucratic  rigidity  in  their  application  and  use,  we  have  moved  closer 
to  that  goal  mentioned  earlier  of  learning  how  to  develop  useful  mathematical 
models  that  can  be  used. 


-4  5- 


5.  BIBLIOGRAPHY 


1.  "Guidelines  for  Documentation  of  Computer  Programs  and  Automated  Data 

Systems:  FIPS  PUB  38,  National  Bureau  of  Standards,  Washington,  DC, 
February  15,  1978. 

2.  Gass,  Saul  I.,  "Computer  Model  Documentation:  A Review  and  an  Approach," 

NBS  Special  Publication  500-39,  National  Bureau  of  Standards, 

Washington,  DC,  February  1979. 

3.  House,  P. , and  McCloud,  J.,  Large-Scale  Models  for  Policy  Evaluation, 

John  Wiley  and  Sons,  New  York,  New  York,  1977. 

4.  "Computer  Program  Documentation  Guidelines,"  NHB-2411-1,  NASA,  Washington, 

DC,  July  1971. 

5.  Lady,  George  M. , "Interim  Model  Documentation  Standards,"  Memo  to  Applied 

Analysis  Senior  Staff,  EIA,  December  4,  19  78. 


APPENDIX  A: 


WORK  STATEMENT 


Energy  Model  Validation  Procedure  Development 


SUMMARY 


This  action  is  an  interagency  agreement  which  initiates  a 
continuing  activity  at  the  National -Bureau  of  Standards 
(NBS)  intended,  to  support  Energy  Information  Administration 
(SXA)  in  model  validation  activities.  The  N3S  is  tasked  to 
make  findings  as  to  the  minimum  model  operating  and  conceptual 
documentation  required  to  conduct  the  model  validation  activities, 
to  examine  model  attributes  and  to  develop  validation  standards 
designed  to  measure  the  ’'confidence”  in  model  results.  For  this 
initial  effort  the  mid-term  oil  and  gas  supply  model  will  be 
the  system  considered.  Particular  model  attributes  to  be 
evaluated  include.:: 


- quality  o: 


data. 


— rational  for  the  model's  logical,  mathematical, 
and  statistical  properties, 

• — methods,  for  comparing  model  results  to  known' 
outcomes,  and  . * • 

— other  characteristics  important  to  developing 
measures  -of  model  confidence. 

Xn-so-far  as  possible',  rigorous  concepts  of  model  confidence 
will  be  developed  and  related  to  the  model  evaluation  process . 
Similar  efforts,  emphasizing  different  EIA  models,  are  planned 
for  the  Los  Alamos  Scientific  Laboratory  (electricity)  .and  the 
MIT  Energy  Laboratory  (coal) 


PROJECT  STATEMENT  OF  WORK 


Title: 


Energy 


Model  Validation  Procedure  Development 


In b r o yUCuro n = This  project  is  part  of  the  Office  of  Analysis 
Overaignu  ar4  Access'  program  to  develop  generic  standards 
anc  procedures  foi:  the  assessment  and  validation  of  Energy 
Information  Administration  (El A)  analysis  systems.  The  EIA 
na^  esmbmsn — many  large  scale  mathematical  and  statistical 
procedures  ~or  Pro3 acting  and  analyzing  energy  production, 
consumption*  prices,  and  associated  impacts.  The  systems  are 
COra?i”x  mathematical,  technological,  and  statistical 
t-chn-gowo  and^ara  lo  be  subjected  to  extensive,  continuing 
re  '/iswo  and^  crz.-p.ques  aimed  at  determining  and  improving 
. tneir  validity,  acpiracy,  and  abilities.  These  systems  will, 
always  be  m a s*_aue  of  chancre.  It  is  imnn-r-r^nt  -For- 


of  each  system  to  determine  its  current  validity,  .and  to 
suggest  improvements  and  state-of-the-art  extensions. 


version 


??■?  ^isns3  have  the  National  Bureau  of  Standards  (NBS) 
establish  a^  continuing  activity  that  will  be  tasked  to  develop 
and  apply  s Lancards _ and  procedures  for  the  validation  of 
analysis  systems  utilized  by  the  EIA  of  the  DOE.  The  goal  of 
this  project  is  to  develop  methods  for  finding  the  degree  of 
confidence  in  results  and  the  circumstances  under  .which  the  * 
systems  may  be  used.  The  procedural  approach  will  be  a 
systematic-. evaluation  of  a system’s  assumptions,  structure, 
i'v*a »_ion , alternatives,  performance,  sensitivity,  and  any 
Ouner  factors  which  are  significant  influences  upon  the 
confidence  in  system  results.  A major  task  will  be  the 
determination,  to  the  extent  possible,  of  rigorous  procedures 
ion  u filming  such  evaluations  in  a determination  of  confidence 
in  system  results  in  specific  uses.  For  the- first  year's  effort 
s oil  and  .gas  supply  foretasting  systems  will  be 

used  m the  development  .of  validation  procedures-  After  the 
. successful  evaluation  of  this  system  additional  systems  will  - 
be  chosen  and  subjected  to  the  evaluations  described  in 
Tasks.  1-3  in  the  scope  of  work. 


The  specific  objectives  of  this  project  are  (1)  to  develop. methods, 

Userut  for  validating  EIA  analysis  systems;  - (2)  to 'establish 

a team  of.  analysts  consisting  of  NBS  personnel  and  outside 
consultants  that  will  accumulate,  and  maintain  expertise  in 
validation  procedures  'as  applied  to  DOE  analysis  systems;  (3) 
initially,  to  evaluate  the  DOE  mid-term  oil  and  gas  supply 
models;  (4)  to  report  to  DOB  on  the  results  of  the  oil.  and  gas 
supply  model  evaluation  and  their  implications  for  users  of 
the  models;  and  .(5)  to  specify  systems  validation  standards 
and  procedures  based  on  the  experiences  of  the  mid-term  oil  and 
gas  supply  model-  evaluation.  'After  the  mid-term  oil  and.  gas 
model  has  been  evaluated  additional  svs tarns  will  be  chosen  by 

" MM ■ 
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The  major  activities  of  this  project  will  be  the  development 
of  system  validation  standards  and  procedures  and  their 
noolication.  to  the  latest  version  of  the  mid-term  oil  and  cja< 
sioplv  analysis  systems,  ' For  the  purposes  of  this  project, 
existing  documentation  reports  will  be  the  initial  input  for 
the  evaluation  process.  It  is  assumed  that  the  systems  and 
their  operating  computer  models  underwent,  during  and  after 
their  initial  developments,  some  verification,  and  validatio: 
tests-  -The-  results  of  any  such  tests,  if  available,  will  be 
valuable  in  x structuring  the  evaluation  to  be  accomplished. 
As  it ’is  not  certain  that  previous  tests  have  been  documents 
this  project  will  establish  both  verification  of  the  program 
and  techniques  of  the  operating  systems-  Thus,  the  proposed 
evaluation  - will  require  the  project  team  to  have  the  means  -t( 
subject  the  systems  to  detailed  scrutiny,  including  program 
listings  and.  the  running  of  the  computer  programs  under  ' 
various  conditions. 


SCOPE  OF  V70RK 


The  following  tasks  will  be  undertaken  by  the  NBS  to  specify 
and  apply  the  validation  procedures. 


Task  Is.-'  Existing  documentation  of  the  oil  and.  gas  analysis 
systems  will  be^  examined 'and  project  personnel  will  esfcablis 
operating  versions,  of  the  systems  for  project  use'. 

Task  2^  ' Operating  and  conceptual  documentation  will  be 
evaluated  and  deficiencies  identified.  For  the  purposes  of 
this  project  a documentation  deficiency  refers  to  any  and 
all  aspects  of  model  documentation- which  is  not  available, 
but  necessary  to' perform  the  other  tasks  of  this  project. 

To  the  extent  to  which  documentation  deficiencies  exist,  sue? 
remedies  as  necessary  to  support  this  validation  project  wil 
be  undertaken-  ‘ It  is  recognized  that  the  extent  of  document 
deficiencies,  if -any,  is  not  now  known.  As  a result,  the 
remaining  tasks  of  -this  project  are  contingent  upon  the 
successful  completion  , of  this  task..  The  resources  allocated 
to  this  cask  in  the  performance  of  the  project,,  including 
the  project  schedule  'for  this  task,  may  differ  from  those 
specified  here  due  to  the  actual  extent  of  documentation 
deficiencies.  / . 
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3 z Systems  attributes  will  be  evaluated  to  include: 


Tar.i  3,1  Completeness  and  accuracy  of  underlying  data, 
Thi: 


sue  fa  sk  calls  lor  a z unciing  or  -che 
sufficiency  cf  the  underlying  data  based  u; 
existing'  documentation  of  the  data?  it  does 
not  call  tor  an  independent  audit  of  any  of  the 
data  at  issue,  • 


Task 


3,2  Conceptual  sufficiency  'of  system  specification. 

This  subtask  calls  for  a finding  as  to  the 
completeness  of  the  set  of  concepts  or  variables 
included  in  the  system  and  the  completeness  of 
the  set  of  interrelationships  among  the  variables 
accounted  for  by  the  model.  Particular  emphasis 
is  placed  .upon  the  identification  of  alternative  . 
specifications  and  the  rationale  for  the  particular 
specification  chosen. 


Task  3.3 'Appropriateness  of  operating- representation. 

This  subtask,  calls  for  a finding  as  to  the  adequacy 
of  the  particular  mathematical  forms  adopted  for  the 
model.  Particular  emphasis  is  placed  on.;' the 
functional  or  algorithmic  forms  employed- for 
determining  variable  values,  alternatives  to -such 
forms  and  the  rationale  for  the  particular  forms 
- chosen  as  well  as  those  rejected-  . . 

Task  3.4  Appropriateness  of  'embodied  estimation  methodologies . 

‘ This  subtask  calls  for  a finding  as  to  the  adequacy- 
of  the-  statistical  or  other  procedures  utilized  to 
derive  the' parameter  values  embodied  in  the  model’s 
mathematical  representation-  Particular  emphasis 
is  placed  "upon  alternative  estimation  procedures 
and  the  rationale  for  the  procedures  selected  for  . 
the  model. 


Task  3.5  System-  sensitivity  ’and  ‘stability.  For  each  of  the 

area3  of  model  attribute  identified  under  spacrf icatior 
representation,  and  estimation  this  subtask  calls  for 
a determination  of  the  sensitivity  or  other  quality, 
of  model  result  associated  with  the  particular 
choices  which  make  up  the  model  itself  compared  to 
the • alternative  choices  not  made.  Particular  emphasis 
is  placed  upon,  a finding  of  the  strengths  and  weax— 
*nesse3  of  the  choices  made  compared  to  therr 
alternatives. 
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rPr'sk  3.6  System  performance  compared  to  known  outcomes. 

This  subtask'  calls  for  the  identification  oi  How 


modeling  results  can  be  verified  by  comparison 
to  known  outcomes  and  how  the  results  of  that 
comparison  can  be  utilized  in  preparing  measure- 
ments or  other  indications  of  confidence  in- model 
results-  If  possible r such  comparisons  will  be 


attempted-  At  a minimum  a methodology  for  making 


and  procedure  for  using  such  comparisons  will  be 
' developed, 


Task  3-7  •.  Computer  related  system  'characteristics  - 


Task"  3. & Any  other  system  element  or  'attribute  which 
• significantly  influences  the  confidence  'in 
. system  results.  . 


Task  4:  The  results  of  the  evaluation,  will  be  consolidated  ^ 

and  a report  on  the  system  strengths  and  weaknesses  prepared!*15 


Task  5 - A specification  of  alternative  concepts  of 
" confidence”  in'  system  results  will  be  prepared. 


Task  6:  A determination  will  be  made  of  the  relationship. 


between  the  outcome  of  the  various  system  attribute  B 

evaluations  and  the  concepts  of  conf idence.  To  the  extent 
possible  a rigorous  statement  of  this  relationship  will  be 
achieved-  • : 


Task  7z  A summary  concept  of  system  result  confidence  will 
be  developed  to' include  the  specification  of -the ‘evaluation 
activities  necessary  to  support 'the  determination  of  system 
result,  confidence.  • • 


Task  8:  An  end  of  year  report  will  be  prepared  on  standards) 

and  procedures  for  determining  system  confidence. 


After  the  successful  evaluation  of  the  mid-term  oil  'and-  gas 
supply  model  additional  systems  will  be  chosen  by- DOE  and 
the  work  recommenced  "'at  Task  1.  ' 


Schedule  (for  first  calendar  year) 

x asK.  I-.-*. - 6 weeks  after  start 

.xask  2 - - - ~ . 12  weeks  after  start 

Tasks  3 and  4 . ....  8 months  after  start 

Tasks  S-fi-7.  an*  « -V>  — U. 
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Reports  for  first;  calendar  year 


TjLj titer  progress  repo__ s.  . 

Interim  Report  on  Tasks  1 

and 

2 

......  3 

months 

after 

start 

Draft  Interim  Report  on 

rp£j 3 and 

..... 

% 

......  S 

months 

af  ter 

s tart 

Interim  Report- on  Tasks  3 

and 

A 

9 

months 

after 

start 

Draft  End  of  Year  Report. 

-----  11 

months 

after 

start 

End  of  Year  Report- - - - 

months 

after 

start 
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"APPROVED  LIST"  MEMO 


/ • 


« — ^ / .i' 


\y 


O.  — >- 

'^ris  o*  * 


UPJ JTED  STATES  DEPARTMENT  OP  CDMP/JERC 
National  Bureau  of  Standards 
Washington.  n.C.  20234 


October  l6,  1978 


ME MORANBUM  FOR  George  Lady 

Office  of  Analytic  Methods 
Department  of  Energy 

sn  >- 

From:  Richard  H.  F.  Jackson 

Operations  Researcn  Division 
National  Bureau  of  Standards 


Subject:  List  of  Documents  Obtained .by  BBS  to  Date 


?nidpt^Lt0  aSf“?  tha*  has  received  all  pertinent  documents 

°,have  for.ths  documentation  evaluation  phase  of  our 
effort,  T»e  provide  belov  the  list  of  documents  received  to  d~te  Tt  ->=? 

“very^t  "^1^  ****”*'  m ° ^ others 

aie  very  draft.  The  list  is  in  no  particular  order. 


(1)  Energy  Information  Administration,  Annual  Report  to  Corigress, 

T ~ _ I ’ - , r?nf  o 10ns . 01  energy-  Supply  and  Demand  and  Their 

Impacts,  April  I978. 


(2)  randS  rT  n&S  S-UPPly  Model>  1977  uPdate>  Research  Memo- 

ranaum  No.  78-015,  December  1977 . 


(3)  J*  lto6?“d  Used  to  Forecast  Domestic  Oil  and  Gas  Sunplv, 

Unpuba-snea  ffo.es  Received  from  C.  Everett,  Undated.'  * " 


Undated!03  0u:t;Line,  0btainad  from  C.  E.  Everett,  Untitled  and 


^ Gd“,  al.ld  Supply  Curves  for  the  Administrator's  Annual  Report 
Technical  Memorandum  TM/ES/78-I7,  September  1978.  ’ 


(6)  Midlevel  Documentation,  ICF  Inc.  Contract  Report,  July  1976. 


(7)  Petrol eu^t  ^ G-, 3 *  S 6 7 8 9“PPly  Co:nI)uter  Program  Documentation,  National 
Petroleum  Council,  November  1973. 


(8)  ' October  W,°m  ^ * *“  °f  °U  “*  GaS  Model*  Received 


(9)  Li  t.  Oi  D..ta  Inputs  to  Oil  and  Gas  Model,  Received  October  10,  1973. 


(10) 
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